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Observing  marine  mammal  distribution  and  determining  population  densities  is  a  complex  problem  that  has  been  ongoing  for  many 
decades.  Modern  approaches  use  a  varying  combination  of  sophisticated  acoustic  monitoring  and  highly  trained  visual  observers  from 
both  ships  and  aircraft.  New  electro-optical  and  infrared  (EO/IR)  camera  systems  with  automatic  detection  algorithms  offer  the  option 
to  do  quantitative  airborne  surveys  safely  and  quickly.  This  paper  discusses  an  airborne  EO/IR  modular  system  used  to  observe  marine 
mammals,  make  density  measurements  and  eventually  be  used  to  quantify  animal  behaviors. 

I.  INTRODUCTION 

Primarily  an  Electro-optical  and  Infrared  (EO/IR)  system  has  been  designed  as  a  cost  effective  modular  sensor  for 
specific  applications  that  can  be  deployed  on  small  Tier  II  unmanned  aerial  systems  (UAS’s).  Over  the  last  few  years 
the  series  of  modular  sensors  (EYE-5f,  EYE-5g  and  EYE-HSswir)  have  been  developed  and  tested  in  a  number  of 
applications.  These  modular  systems  will  not  require  additional  data  link  bandwidth  nor  would  they  add  to  the  already 
heavy  workload  of  the  data  analyst.  The  systems  have  shown  that  they  can  add  value  in  border  patrol,  search  and 
rescue  and  littoral  reconnaissance  operations  and  can  make  real-time  detections  and  data  products  of  submerged  targets 
(submarines,  semi-submersible,  floating  mines,  etc.)  in  maritime  operations. 

The  system  for  marine  mammal  detection,  called  EYE-5f,  is  a  coupled  sensor  head  with  Global  Positioning  System 
(GPS)  and  an  image  processing  system.  The  sensor  head  is  a  turreted  package  containing  a  Long  Wavelength  Infrared 
(LWIR)  camera  for  night  operations,  a  three-channel  multispectral  system  for  animal  detection  and  a  32X  video  zoom 
camera  to  aid  in  classification  and  identification.  The  processor  is  a  Graphics  Processing  Unit  (GPU)  based  system 
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capable  of  producing  real-time  processed  imagery  within  100  milliseconds  of  acquisition.  Field  data  collections  using 
this  system  and  resulting  quantitative  calculations  are  presented. 

Visual  surveys  from  ships  are  generally  done  using  standard  line  transect  techniques.  Transect  lines  are  established  prior 
to  the  survey  to  uniformly  cover  a  predefined  study  area.  Marine  mammals  are  spotted  by  a  team  of  observers  using 
both  standard  (7X)  and  high  power  (25X)  binoculars.  When  the  animal  or  group  is  seen,  the  location  of  the  animal  or 
group  is  determined  by  recording  their  relative  bearing  to  the  bow  of  the  ship  and  their  distance  using  the  reticules 
in  the  binoculars.  From  aircraft,  the  location  of  the  animal  is  determined  by  using  a  circle-back  method  to  determine 
the  animal’s  location.  In  both  ship  and  aircraft  surveys,  environmental  conditions  are  routinely  recorded  to  adjust  for 
visibility  conditions  in  density  calculations.  Quantitative  density  and  abundance  estimates  are  made  using  ‘DISTANCE 
sampling’  methodology,  which  accounts  for  the  portion  of  the  animals  that  were  not  detected  near  the  track  line  and  are 
not  on  the  surface.  During  a  recent  field  test  in  Hawaii,  it  was  demonstrated  that  using  EYE-5f,  animals  were  detected 
in  excess  of  40  meters  in  depth  (clear  water)  in  sea  states  of  5  and  greater.  This  both  increases  the  number  of  animals 
detected  (reducing  uncertainty)  and  reduces  the  sensitivity  of  probability  of  detection  due  to  environmental  conditions 
when  compared  to  visual  techniques. 

II.  IMAGING  AND  PROCESSING  SYSTEMS 

The  EYE-5f  modular  multispectral  imaging  system  designed  by  Advanced  Coherent  Technologies  (ACT),  LLC  is 
shown  in  Fig.  1.  The  system  dimensions  are  presented  in  Table  1.  It  is  composed  of  three  multispectral  cameras  with 
narrowband  interference  filters  optimized  to  allow  the  system  to  detect  marine  mammals  beneath  the  water  surface,  a 
Sony  32X  zoom  video  camera  and  a  640x480  pixel  long  wave  infrared  (LWIR)  camera.  These  cameras  are  integrated 
into  a  Cloud  Cap  Tase300  turret.  The  turret  enables  the  system  to  scan  laterally  to  40°  from  nadir  on  either  side  of  the 
aircraft  as  it  flies.  As  it  scans,  the  multispectral  cameras  search  for  whales  in  the  water  column  down  to  depths  as  deep 
as  50  meters  in  clear  water.  Once  detected,  the  system  can  lock  on  and  track  the  animal  until  it  surfaces,  where  high- 
resolution  pictures  from  the  zoom  video  system  can  be  used  by  the  observer  to  identify  species  and  possibly  photo 
identify  individuals.  The  LWIR  system  can  also  be  used  at  night  to  detect  blowing  or  breaching  animals  off  nadir,  where 
the  emissivity  of  the  animal  is  much  greater  than  the  water.  Computational  power  for  EYE-5f  system  is  provided  by  the 
Vision4ce©  General-purpose  Ruggedized  Integrated  Processor  (GRIP)  Delta  and  is  also  shown  in  Fig.  1. 


Fig.  1  -  EYE-5f  in  display  mount  on  the  left  and  data  acquisition/processing  system  Vision4ce  Grip 

Delta  on  the  right 
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Table  1  -  EYE-5f  sensor  dimensions 


Overall  System 

Size: 

15  2  x  178  x  229  mm  (6x7x9  inches) 

Turret  Diameter: 

178  mm  (7.0  inches) 

Camera  Payload  Volume: 

125  in3 

Weight: 

Under  100  lbs 

The  GRIP  Delta  is  a  Commercial  Off-The-Shelf  (COTS)  rugged  computer  system  specifically  designed  for  applications 
that  can  take  advantage  of  GPU  processing.  The  basic  hardware  for  the  system  includes  an  Intel©  Core  2  Duo  Processor, 
4GB  of  system  memory,  a  256GB  Solid-State  Drive  (SSD)  for  storage  and  an  NVIDIA®  GT240M  graphics  processor. 
Additionally,  there  is  expansion  space  available,  via  PCIe  slots,  that  allow  for  additional  storage,  standard  input/output 
(I/O),  or  custom  I/O  ports  such  as  Camera  Link,  HD  video,  etc.  The  COTS  design  allows  ACT  to  easily  upgrade  any 
and  all  components  as  they  become  available  and  ensure  that  there  is  robust  support  for  all  of  the  hardware  featured  in 
the  system. 

The  EYE-5f  system  outputs  three  data  streams  from  five  cameras.  The  data  streams  from  the  LWIR  camera  and  the 
Sony  zoom  video  camera  are  formatted  for  both  a  video  tracker  and  a  downlink.  Further  processing  on  the  ground  is 
possible  if  the  downlink  is  available.  The  third  is  the  multi-camera  data  stream  that  is  transferred  from  the  turret  to 
the  processor  with  wideband  optical  fibers.  This  multispectral  data  stream  requires  a  series  of  processing  steps.  These 
steps  are  necessary  to  exploit  the  data  so  that  subsurface  animals  are  visible  and  automatically  detected.  These  general 
processing  steps  are  shown  in  Fig.  2. 


Fig.  2  -  Data  flow  diagram  for  general  processing  of  multispectral  data 


The  addition  of  the  NVIDIA  GPU  significantly  increases  the  amount  of  computational  power  available  in  the  system 
through  the  use  of  Compute  Unified  Device  Architecture  (CUD A)®.  This  architecture  allows  for  highly  parallel 
computations  utilizing  thousands  of  threads  among  many  cores,  a  structure  that  is  ideal  for  performing  image-processing 
algorithms.  These  include,  but  are  not  limited  to,  real-time  marine  mammal  detection,  scene  tracking  and  stabilization, 
image  enhancement,  or  other  numerically  intensive  vector  algorithms.  This  provides  the  EYE-5 f  system  with  the  ability 
to  perform  many  of  these  image-processing  functions  onboard  in  real  or  near  real  time  rather  than  in  post  processing. 
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Also,  by  placing  the  computations  on  the  GPU  rather  than  the  CPU,  there  is  typically  a  speedup  of  multiple  orders  of 
magnitude  (algorithm  dependent)  as  well  as  freeing  CPU  resources  to  deal  with  such  tasks  as  image  acquisition,  storage, 
user  input  and  control  of  any  additional  peripherals  (such  as  a  GPS)  that  might  be  involved.  NVIDIA  CUDA  is  also  a 
COTS  product  and  carries  with  it  a  great  deal  of  support  and  documentation,  making  it  ideal  in  a  low  cost,  flexible  and 
powerful  image  processing  framework. 

III.  MARINE  MAMMAL  DETECTION  AND  DENSITIES 

ACT  has  been  developing  ways  to  increase  the  Signal-to-Noise  Ratio  (SNR)  of  whale  signatures  at  depth  with  electro- 
optical  systems.12  For  this  effort  ACT  has  designed  a  detection  algorithm  that  utilizes  both  the  spectral  and  the  spatial 
domains. 

Spectral  Processing 

The  EYE-5f  in  maritime  application  uses  three  multispectral  cameras  with  narrow  band  interference  filters,  a  blue  band 
near  480  nm  with  a  20  nm  bandpass,  a  green  band  near  535  nm  with  a  20  nm  bandpass  and  a  red  band  near  600  nm  with  a 
10  nm  bandpass.3  The  assumption  is  that  the  ability  to  see  into  the  water  column  is  limited  by  the  spatially  variable  light 
reflected  from  the  sea  surface.  This  variable  light  is  fairly  wideband  with  blue  through  red  light  reflected  from  the  surface 
in  constant  proportions.  The  correlation  between  these  three  bands  is  used  in  the  spectral  processing  by  subtracting  a 
weighted  difference  between  the  blue  band  and  the  green  band.  The  weighting  is  determined  by  minimizing  the  variance 
of  the  resulting  image.  The  red  band  is  used  primarily  to  eliminate  residual  sea  surface  structure  such  as  cloud  shadows, 
kelp  rafts,  etc. 

Spatial  Processing 

In  the  spatial  domain  the  processing  utilizes  two  main  components:  an  inverse  covariance  matrix  and  a  signal  vector. 
The  inverse  covariance  matrix  is  used  to  build  an  understanding  of  the  spatial  relationship  within  the  dimensions  of  the 
signal  vector.  The  signal  vector  is  looking  for  a  grouping  of  target  pixels,  depicted  in  Fig.  3  indicated  by  the  l’s  in  the 
circled  area.  Combining  this  information  into  a  spatial  matched  filter  and  applying  it  to  the  spectrally  processed  data 
results  in  maximizing  the  SNR  of  the  imagery. 
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Fig.  3  -  Graphical  representation  of  the  signal  vector 
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Whale  Detection  Algorithm 

The  whale  detection  algorithm  can  best  be  described  by  watching  how  an  image  transforms  as  it  goes  through  these 
different  steps.  The  image  used  for  Figs.  4  through  7  was  taken  just  west  of  Maui,  Hawaii,  in  February  2009  at  an 
altitude  of  2000  feet.  The  configuration  used  for  the  multispectral  cameras  was  a  blue  band  (485  nm)  and  a  green  band 
(529  nm)  with  23  mm  lenses.  There  is  a  large  difference  in  light  levels  between  the  center  of  the  image  and  at  the  edges 
before  the  processing  as  shown  in  Fig.  4(a).  This  is  primarily  due  to  the  lens  and  sun  effects.  It  has  been  estimated 
that  this  is  a  quadratic  trend  in  the  image  and  a  least-squares  estimator  is  used  to  remove  this  effect.  This  technique  is 
effective  and  practical  because  it  extracts  the  data  necessary  from  the  image  with  no  need  for  a  priori  information.  An 
example  of  the  estimated  trend  can  be  seen  in  Fig.  4(b). 


Fig.  4(a)  -  Unprocessed  image  Fig.  4(b)  -  Estimated  quadratic  trend 


Figure  5  shows  the  image  after  the  trend  has  been  removed.  Notice  how  the  edges  are  no  longer  dark.  This  sets  up  the 
detector  with  a  great  understanding  of  what  to  expect.  The  algorithm  will  now  be  able  to  quantify  how  much  deviation 
from  normal  there  is  for  a  whale  like  object  and  classify  it  as  a  target  or  not. 


Blue-trend 
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Fig.  5  -  Trend  removed 
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The  spectral  process  performed  requires  the  trend  to  be  removed  from  both  the  blue  and  the  green  band.  Once  this  is 
done,  the  spatial  process  is  applied  and  is  shown  here  as  a  step  in  the  overall  process.  Notice  in  Fig.  6  that  a  majority  of 
the  sea  surface  waves  have  been  removed  compared  to  Fig.  5.  This  reduction  is  expected  since  the  spectral  process  is 
removing  the  sea  surface  clutter.  A  line  profile  was  added  and  the  image  was  annotated  with  the  calculated  SNR  gain  of 
3.5  dB  associated  with  the  spectral  processing. 
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Fig.  6  -  Spectrally  processed  image.  The  deep  submerged  whale  is  shown  in  the  circled  area. 


The  last  image-enhancing  technique  that  is  utilized  is  in  the  spatial  domain.  Figure  7  was  created  using  a  7x7  size 
mask  convolved  over  the  image  in  Fig.  6.  This  mask  is  a  good  all-purpose  filter  for  whales  at  depth  and  is  proving  very 
effective.  The  size  of  the  mask  can  vary.  It  depends  on  a  target  size.  Notice  the  whale  just  to  the  right  of  the  main  pod  in 
Fig.  7,  which  was  virtually  impossible  to  see  in  Fig.  6  before  the  spatial  processing  was  applied. 
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Fig.  7  -  Spatial  and  spectral  processing  applied 


Estimating  Marine  Mammal  Population  Density 

All  data  collected  used  the  EYE-5  f  system  and  were  aimed  at  collecting  data  necessary  to  insure  that  the  automatic 
detection  algorithms  were  working  properly  and  in  real  time.  An  overall  goal  was  to  assess  the  use  of  the  EO/IR  system 
to  make  marine  mammal  density  measurements.  In  order  to  meet  this  goal,  ACT  developed  a  plan  for  making  aerial 
surveys  with  the  EYE5  system.  The  aerial  survey  process  includes  the  following  steps: 

1 .  Conduct  a  strip  transect  survey.  The  strips  should  be  spaced  to  insure  that  an  animal  detected  in  one  pass  cannot 
swim  to  the  next  before  the  aircraft  gets  there.  This  insures  no  animal  is  counted  twice. 

2.  Fly  tracks  that  are  not  influenced  by  the  presence  of  visual  detections.  The  purpose  of  this  is  that  it  is  assumed 
each  observation  is  independent  and  random.  This  allows  us  to  get  the  density  by  simply  dividing  the  number 
of  animals  detected  (adjusted  to  compensate  for  the  dive  profile  and  the  water  clarity)  by  the  area  surveyed  to 
get  the  density.  If  visual  observations  were  taken  into  account,  the  cross  track  probability  of  detection  for  the 
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visual  observer  would  need  to  be  accounted  for.  This  would  increase  the  complexity  of  the  density  calculation. 

3.  Estimate  the  dive  profile  of  the  type  of  whales  (in  this  case  Humpbacks)  to  get  the  probability  of  seeing  an 
animal  above  the  depth  to  which  the  sensor  can  detect.  While  this  may  not  be  available  for  many  species,  it 
is  essential  information  for  any  type  of  density  survey  and  must  be  estimated.  This  reinforces  the  valuable 
information  collected  by  tagging. 

4.  Quantify  EYE-5f’s  depth  capability  as  determined  by  water  clarity,  surface  waves,  sun  glint,  etc. 

5.  Calculate  the  area  covered  during  the  survey. 

6.  Count  the  number  of  whales  detected. 


The  density  (D)  is  then  given  by  the  equation: 


(i) 


where  N  is  the  number  of  whales  counted,  A  is  the  area  covered  by  the  camera  images  and  P  is  the  probability  that  if  a 
whale  is  present  in  the  field  of  view  it  is  detected. 


In  March  2010,  the  first  density  survey  was  undertaken  over  the  southwestern  waters  of  Maui,  Hawaii.  This  area  is 
breeding  and  calving  ground  for  Humpback  whales  in  the  winter  months.  During  this  time,  the  density  of  the  animals 
is  fairly  high  and  is  perfect  for  testing  the  utility  of  using  EYE-5  f  to  determine  animal  density  using  the  electro-optical 
system.  A  less  than  two-hour  collect  was  used  to  measure  whale  densities  from  an  altitude  between  1500  and  2000  feet. 
Altitude  was  limited  by  cloud  cover  and  not  by  system  capability.  For  large  whales,  the  system  can  easily  be  operated 
in  excess  of  5000  feet  in  clear  skies.  Figure  8  shows  the  tracks  flown  in  the  survey.  While  these  actual  tracks  are  only 
generally  similar  to  the  planned  tracks  to  be  flown  in  the  strip  survey,  they  are  still  not  influenced  by  the  presence  of 
animals  and  thus  detections  are  still  random  and  the  survey  is  valid.  A  sea  state  of  6  was  recorded  with  wind  gusts  greater 
than  40  knots  at  times.  The  high  winds  are  the  primary  reason  that  the  actual  tracks  flown  do  not  resemble  a  regular,  and 
planned,  grid.  Under  these  high  wind  conditions,  visual  airborne  surveys  would  be  impossible. 
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Fig.  8  -  Google  image  of  day  one  afternoon  flight  tracks  for  two-hour  whale  survey 


To  build  a  distance  sampling  survey,  the  dive  profile  of  a  Humpback  whale  needs  to  be  quantified.  The  information 
used  can  be  seen  in  Table  1 .4  Table  2  is  derived  from  Baird’s  measurements  of  dive  profiles  for  a  limited  number  of 
Humpbacks  in  the  same  area  as  our  survey.  It  shows  the  average  cumulative  percentage  of  time  the  animals  spend  above 
a  specific  depth.  This  cumulative  depth  distribution  would  technically  be  valid  only  for  behaviors,  sex  and  sizes  of  the 
whales  represented  in  the  Baird’s  measurements.  The  next  step  is  to  quantify  EYE-5f’s  ability  to  detect  at  depth.  It  has 
been  verified  that  EYE-5f  can  detect  in  excess  of  1.5  diffuse  attenuation  lengths  in  Hawaiian  waters  (Jerlov  water  type 
lb  under  the  conditions  of  these  data  collects),  or  approximately  forty  meters.5  Comparing  this  to  the  dive  profile  of  a 
Humpback  whale,  there  is  an  82%  probability  that  EYE-5f  will  detect  a  Humpback  whale  in  this  region  if  one  is  in  the 
sensor’s  field  of  regard. 
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Table  2  -  Baird’s  Dive  Profile  of  Humpback  Whale  in  the  Maui  Breeding  and  Calving  Areas 


Depth  (meters) 

Mean  %  time  in  Depth  Category 

Cumulative  %  time 

0-10 

39.55 

39.55 

11-20 

26.51 

66.06 

21-30 

11.65 

77.71 

31-40 

4.25 

81.96 

41-50 

3.04 

85.00 

51-60 

2.47 

87.47 

61-70 

2.14 

89.61 

71-80 

1.66 

91.27 

81-90 

1.97 

93.24 

91-100 

1.55 

94.79 

An  example  of  the  enhancement  for  detection  that  the  ACT  algorithm  can  provide  is  shown  in  Figs.  9  and  10.  This  image 
was  taken  with  the  EYE-5f  system  in  sea  state  6  and  contains  two  Humpback  whales.  The  whales  are  difficult  to  see  in 
the  preprocessed  image  in  Fig.  9,  but  after  the  band  difference  minimizing-variance  algorithm,  shown  in  Fig.  10,  they 
are  clearly  visible  and  ready  for  spatial  processing. 


Fig.  9  -  Multispectral  pre-processed  data  with  two  Humpbacks 
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Fig.  10  -  Multispectral  post-processed  image  data  clearly  showing  the  two  Humpbacks 


The  next  step  in  the  survey  process  is  to  determine  the  number  of  whales  detected  and  the  survey  area  covered.  The 
processed  survey  data  revealed  20  whales  detected  in  13  “pods”,  or  groups,  and  covered  a  survey  area  of  approximately 
30  square  miles.  There  was  an  average  of  1 .54  whales  per  pod.  The  equation  for  Humpback  whale  density  was  presented 
in  Eq.  1.  The  estimated  Humpback  whale  density  for  the  area  covered  is  0.32  Humpback  whales  per  square  kilometer. 

If  it  is  assumed  that  pods  of  whales  do  not  interact  with  each  other,  then  a  Poisson  distribution  in  pods  (rather  than 
individual  whales)  can  be  used  to  estimate  uncertainty.  The  error  in  number  of  whales  is  thus  the  square  root  of  the 
number  of  pods  times  the  number  of  whales  per  pod,  or  about  six  whales.  The  density  is  thus  0.32  whales  (+/-  0.089) 
whales  per  square  kilometer.  In  the  roughly  400  square  miles  between  Maui  and  Lanai  and  in  Maui’s  Ma’alaea  Bay, 
there  would  thus  be  about  333  Humpback  whales  (+/-  92  whales)  during  the  winter  breeding  /  calving  months  assuming 
a  uniform  spatial  distribution  over  this  area.  Either  increasing  the  swath  or  increasing  the  amount  of  time  taken  to 
conduct  the  survey  can  minimize  the  error  of  estimation.  Both  could  easily  be  done.  It  should  be  noted  that  even  with  the 
narrow  field  of  view  of  the  EYE-5f  multispectral  system,  a  survey  covering  30  square  miles  takes  less  than  two  hours. 

Over  the  past  several  years,  the  Pacific  Whale  Foundation  (PWF)  has  been  organizing  volunteers  to  count  the  number 
of  Humpback  whales  visibly  in  the  same  area  of  the  EYE-5f  airborne  surveys.  The  sighting  methodology  is  consistent 
from  year  to  year  so  the  counts  can  reflect  relative  population  changes  over  time.  The  sighting  average  over  the  last  few 
years  is  about  1300,  with  1208  sightings  in  2010.  The  2010  counts  were  done  just  three  days  prior  to  EYE-5f  surveys; 
however,  their  counts  are  conducted  from  survey  points  all  around  Maui  and  not  just  from  Lahaina  thru  Ma’alaea  Bay 
area  as  surveyed  by  EYE-5 f.  Although  the  sighting  methodology  is  consistent  year  to  year,  the  animals  are  potentially 
counted  multiple  times  by  multiple  people  and  the  data  is  thus  not  a  direct  density  measurement. 

The  aerial  survey  demonstrated  that  this  type  of  automated  system  could  be  very  useful  in  quantitatively  measuring 
marine  mammal  densities  quickly  when  the  species  densities  are  reasonably  high.  For  an  accurate  survey,  it  can  be 
assumed  that  at  least  20  detections  are  needed.  More  detections  decrease  the  numerical  uncertainty,  and  less  detections 
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increase  it.  During  the  survey,  the  system  did  not  scan,  so  the  field  of  regard  was  just  10°  either  side  of  nadir.  If  scanning 
is  employed,  the  field  of  regard  can  be  increased  by  a  factor  of  four  to  plus  and  minus  40  degrees  from  nadir  and  still 
get  the  same  detection  depth  performance.  In  addition,  the  altitude  can  be  doubled  to  4000  feet  easily  and  still  have 
the  resolution  needed  for  detecting  animals  as  small  as  dolphins.  Those  two  factors  coupled  with  an  increase  in  aircraft 
speed  gives  an  order  of  magnitude  increase  in  area  search  rate;  meaning  that  in  the  same  two  hour  survey,  200  whales 
would  have  been  sighted,  or  20  whales  would  have  been  sighted  if  the  density  was  10  times  smaller.  Ultimately  the 
time  required  to  conduct  a  survey  depends  on  the  density  of  the  animal  population  and  the  acceptable  error.  The  EYE-5  f 
system  is  designed  to  make  the  best  use  of  this  time,  so  that  it  can  operate  in  relatively  low  densities  and  still  provide 
accurate  findings. 

IV.  MARINE  MAMMAL  BEHAVIOR 

Overall  uncertainties  are  again  dominated  by  the  uncertainty  of  the  marine  mammal  distribution  model.  In  this  case, 
proper  dive  profiles  and  possibility  of  seeing  the  whales  at  depth  are  again  very  important  to  correctly  normalize  the 
counts.  The  designed  detection  algorithm  has  enhanced  the  image  to  the  point  where  targets  below  the  sea  surface  show 
up  that  were  not  previously  seen.  The  following  examples  highlight  the  fact  that  the  EYE-5 f  multispectral  imaging 
system  can  see  beneath  the  surface.  This  capability  has  many  implications,  including  the  potential  to  improve  the 
accuracy  of  distance  sampling  surveys. 

Depth  of  Whales 

The  algorithm  utilizes  a  depth  equation  that  uses  the  contrast  difference  between  a  whale  at  the  surface  and  a  whale  at 
depth.  The  equation  is: 


z  =  ln[C(z)/C(0)]/(2k),  (2) 

where  z  is  the  depth  of  the  target  measured  in  meters,  C(z)  is  the  contrast  of  the  target  at  depth  z,  C(0)  is  the  contrast  of 
the  target  at  the  surface  and  k  is  the  downwelling  irradiance  diffuse  attenuation  coefficient  with  units  of  1 /meters,  k  can 
be  estimated  using  previous  water  clarity  measurements  or  eventually  directly  from  the  data  itself  based  on  the  color  of 
the  water.  This  equation  will  provide  the  data  necessary  to  determine  our  system’s  capability  to  determine  the  depth  of 
whales  detected.  Figures  11  through  14  are  presented  examples  of  whale  depth  calculations.  Note  that  it  is  not  necessary 
that  the  C(0)  whale  be  in  every  image,  as  long  as  the  image  in  which  the  zero  depth  whale  contrast  was  measured  was 
taken  under  the  same  conditions. 


Fig.  11  -  Processed  image  of  nine  whales  detection  at  depth  (z)  on  the  right  with  the  corresponding  RGB  image  on  the  left 
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Fig.  13  -  Processed  image  of  three  whales  detection  at  depth  (z)  on  the  right  with  the  corresponding  RGB  image  on  the  left 


Fig.  14  -  Processed  image  of  two  whales  detection  on  the  right  with  the  corresponding  RGB  image  on  the  left.  The  processed 
image  indicates  a  whale  at  depth  just  to  the  right  of  the  obvious  whale  image. 
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Size  Distribution  and  Distance  Between  Whales 

In  order  to  determine  a  whale  size  distribution,  images  of  12  whales  were  taken  (10  adults  and  2  calves).  Sizes  were 
measured  in  meters  using  the  ruler  tool  in  Google  earth.  The  average  size  of  an  adult  whale  is  12.9  meters,  and  a  calf 
(baby  whale)  is  7.7  meters. 

Figure  15  presents  an  example  of  Google  image  showing  one  of  the  processed  images  with  a  pod  (2  whales)  of 
Humpbacks.  The  process  uses  the  Euler  angles  of  the  gimbal  and  the  aircraft  along  with  the  fixed  field  of  view  of  the 
imager  to  geolocate  the  four  corners  of  the  image  on  the  sea  surface.  Once  geolocated,  apparent  sizes  and  distances  can 
be  readily  measured  with  less  than  1  meter  accuracy. 
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Fig.  15  -  Google  image  showing  one  of  the  processed  images  with  a  pod  of  Humpbacks.  The  image  is  clamped  to  the  sea 

floor  on  Google  Earth  to  measure  the  size  of  the  whales. 


Figure  16  presents  an  example  of  image  showing  3  Humpback  whales  and  example  of  calculation  whale  depth  (z), 
whale  size  (s)  and  distance  between  the  whales  (d).  The  images  from  marine  mammal  survey  were  positioned  in  the 
Google  Earth  map  and  clamped  to  the  sea  floor  in  order  to  measure  size  of  whales  and  distance  between  whales.  The 
whale  depth,  size  and  distance  between  whales  are  measured  in  meters. 
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Fig.  16  -  Example  of  data  showing  whale  depth  (z),  whale  size  (s)  and  distance  between  whales  (d) 


All  images  collected  during  marine  mammal  densities  survey  come  along  with  metadata,  which  was  used  to  annotate 
the  imagery.  Definitions  of  the  annotations  are  shown  in  Fig.  17.  The  annotation  can  be  used  to  place  the  image  in  space 
and  time  as  well  as  provide  the  geo  location  of  the  camera  and  define  the  collection  angles;  this  allows  further  analysis 
of  the  image.  In  the  marine  mammal  monitoring  case,  further  analysis  has  included  marine  mammal  size,  depth,  spatial 
separation  and  density. 
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Fig.  17  -  Definitions  of  the  annotation  (meta  data)  for  the  image  presented  in  Figure  16 


V.  CONCLUSIONS 

A  reasonably  low-cost,  multispectral,  multichannel,  turreted  optical  system  has  been  presented  that  is  suitable  for  use 
on  small  UAS  platforms  as  well  as  manned  fixed  and  rotary  wing  aircraft.  The  system  has  been  shown  to  be  capable  of 
making  quantitative  observations  of  Humpback  whales  both  on  and  near  the  surface  of  the  ocean  in  a  known  breeding 
ground.  When  the  marine  mammal  densities  are  sufficient  (greater  than  about  1  animal  per  hundred  square  miles),  this 
type  of  system  can  provide  a  quick  and  efficient  way  to  remotely  measure  the  animal  density  for  the  surveyed  region. 
The  system  can  also  be  used  to  monitor  the  effects  of  nearby  ship  traffic  and  airborne  observers  and,  if  timing  is  good, 
to  record  how  the  animals  directly  react  to  other  nearby  anthropogenic  sounds.  The  key  to  monitoring  behaviors  is  the 
ability  to  observe  the  animal  over  a  period  of  time.  To  do  this,  the  aircraft  could  easily  orbit  a  high  enough  altitude  to 
presumably  be  unnoticed  by  the  animals  and  still  be  constantly  observing  and  quantifying  depths  and  relative  positions. 

While  this  paper  presents  a  system  used  only  for  observing  the  behaviors  and  densities  of  underwater  animals,  there  are 
clearly  other  application  for  this  type  of  system.  EO/IR  multispectral  systems  have  been,  and  are  being,  used  in  a  variety 
of  maritime  applications  including  search  and  rescue,  mine  detection  and  antisubmarine  warfare. 
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